Electronic Device and Method for Image Stabilization

ABSTRACT

An electronic device and methods for image stabilization for use in an electronic device are provided. The method includes the steps of: receiving a plurality of motion data, at least a portion of the plurality of motion data corresponding to a first image frame, determining a motion value corresponding to the first image frame according to the portion of motion data, determining a display offset between the first image frame to a reference image frame according to the motion value and providing a portion of the first image frame for display according to the display offset. The reference image frame is displayed prior to the first image frame and is stored in a storage unit.

This application claims the benefit of U.S. Provisional Application Ser.No. 61/602,690 filed on Feb. 24, 2012.

CROSS-REFERENCES TO RELATED APPLICATIONS

Not applicable.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an electronic device and methods forimage stabilization for use in an electronic device; more specifically,the present invention relates to an electronic device and methods forimage stabilization that utilize a plurality of sensor data fordetermining motion related information and the consequent operations.

2. Descriptions of the Related Art

Image alignment is a technique used for geometrically matching twoimages according to their contents for certain purposes such as erasinghand shaking when taking multi-pictures. Image alignment can be appliedto various multi-frame image processing domains such as high dynamicrange preprocessing, video stabilization, multi-frames noise removal,multi-frames object removal, panorama stitching, etc.

For certain real-time applications like on-device video stabilizationfor video recording, computing performance becomes critical whenaligning two neighboring frames. Hence, an efficient and accuratetechnique for image/video stabilization, especially for on-devicestabilization, is in an urgent need.

SUMMARY OF THE INVENTION

An objective of this invention is to provide a method for imagestabilization for use in an electronic device. The method comprises thefollowing steps of: (a) capturing an image frame, (b) receiving aplurality of sensor data during a time interval comprising a time theimage frame is captured, (c) determining a motion variance correspondingto the image frame according to the plurality of sensor data, and (d)determining whether to perform a stabilization on the image frameaccording to the motion variance. In response to the stabilization beingdetermined, the method further comprises the following steps of:determining a motion offset of a saliency region of the image frame to areference saliency region of a reference image frame, cropping a portionof the image frame according to the motion offset, and displaying theportion of the image frame on a display unit of the electronic device.It is noted that the reference image frame is captured prior to theimage frame and is stored in a memory unit within the electronic device.

Another objective of this invention is to provide an electronic device.The electronic device comprises an image input unit, a sensor inputunit, an image processing unit, and a display unit. The image processingunit comprises a motion analysis unit, a motion determination unit, anda stabilization unit. The image input unit is configured to capture animage frame. The sensor input unit is configured to receive a pluralityof sensor data, wherein the plurality of sensor data is received by thesensor input unit during a time interval comprising a time the imageframe is captured. The image processing unit is configured to processthe image frame. The motion analysis unit is configured to determine amotion variance corresponding to the image frame according to theplurality of sensor data. The motion determination unit is configured todetermine whether to perform a stabilization on the image frameaccording to the motion variance corresponding to the image frame. Thestabilization unit is configured to determine a display offset of theimage frame with respect to a reference image frame according to thestabilization determination. The display unit is configured to display aportion of the image frame according to the display offset.

A further objective of this invention is to provide a method for imagestabilization for use in an electronic device. The method comprises thesteps of: (a) receiving a plurality of motion data, at least a portionof the plurality of motion data corresponding to an image frame, (b)determining a motion degree corresponding to the image frame accordingto the portion of motion data, (c) determining a display offset betweenthe image frame to a reference image frame according to the motiondegree, and (d) providing a portion of the image frame for displayaccording to the display offset. The reference image frame is displayedprior to the image frame and is stored in a storage unit.

The present invention determines motion information (e.g. theaforementioned motion variance, motion value, and/or motion degree) ofan image frame according to sensor data that are received within a timeinterval comprising a time the image frame is captured, determineswhether stabilization is required, and determines a display offset forthe image frame. Various kinds of motion information are provided by thepresent invention so that whether the stabilization is required can bedetermined more accurately and efficiently. In addition, various kindsof techniques for performing motion estimation are provided by thepresent invention so that the display offset can be determined moreaccurately and efficiently. Hence, the effect of small disturbancescaused due to undesired shaking (e.g. hand shaking) can be minimized

The detailed technology and preferred embodiments implemented for thesubject invention are described in the following paragraphs accompanyingthe appended drawings for people skilled in this field to wellappreciate the features of the claimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an electronic device 1 of the first embodiment;

FIG. 1B illustrates a portion 100 d of the image frame 100, the displayoffset 110, and the portion 102 d of the image frame 102;

FIG. 2 illustrates an electronic device 2 of the second embodiment;

FIG. 3A illustrates an electronic device 3 of the third embodiment;

FIG. 3B illustrates the concept of determining the motion ratio 109;

FIG. 4A illustrates an electronic device 4 of the fourth embodiment;

FIG. 4B illustrates the stabilization unit 455;

FIGS. 5A-5E illustrates the flowcharts of the fifth embodiment; and

FIG. 6 illustrates the flowchart of the sixth embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description, the present invention will be explainedwith reference to embodiments thereof. However, these embodiments arenot intended to limit the present invention to any specific environment,applications, or particular implementations described in theseembodiments. Therefore, description of these embodiments is only forpurpose of illustration rather than to limit the present invention. Itshould be appreciated that, in the following embodiments and theattached drawings, elements unrelated to the present invention areomitted from depiction.

A first embodiment of the present invention is an electronic device 1and a schematic view of which is depicted in FIG. 1A. The electronicdevice 1 comprises an image input unit 11, a sensor input unit 13, animage processing unit 15, and a display unit 17. The image processingunit 15 comprises a motion analysis unit 151, a motion determinationunit 153, and a stabilization unit 155. The motion analysis unit 151 iselectrically connected to the sensor input unit 13 and the motiondetermination unit 153. The stabilization unit 155 is electricallyconnected to the motion determination module 153, the image input unit11, and the display unit 17. The display unit 17 is further electricallyconnected to the image input unit 11.

The image input unit 11 may be a web camera or any other device that isable to capture or provide image frames. The sensor input unit 13 may bea G-sensor, an accelerometer, a gyro meter, or any other sensor that isable to generate a plurality of sensor data that is related to themovement of the electronic device 1. The image processing unit 15 may beany of various processors, central processing units (CPUs),microprocessors, or other computing devices that are well-known by thoseof ordinary skill in the art.

Since the present invention is related to image stabilization, thefollowing descriptions will be focused on the process for imagestabilization when the electronic device 1 is used for taking a video ora sequence of image frames.

Within a time interval, the image input unit 11 captures an image frame102 and the sensor input unit 13 receives a plurality of sensor data104. It is noted that the sensor data 104 may be provided at a constantrate. Preferably, the constant rate is higher than or equal to an imageframe rate provided by the image input unit 11.

Afterwards, the image processing unit 15 processes the image frame 102in terms of image stabilization. The motion analysis unit 151 determinesa motion variance 106 corresponding to the image frame 102 according tothe sensor data 104. Since both the image frame 102 and the sensor data104 are derived within the same time interval, the motion variance 106reflects the condition/degree of the movement of the electronic device 1as well as the image frames (including the image frame 102) captured bythe electronic device 1 within the time interval. Therefore, the motiondetermination unit 153 then determines whether to perform stabilizationon the image frame 102 according to the motion variance 106corresponding to the image frame 102.

Next, the stabilization unit 155 determines a display offset 110 of theimage frame 102 with respect to another image frame 100 according to thedetermination result 108. It is noted that the image frame 100 iscaptured before the image frame 102 by the image input unit 11 and istreated as a reference image frame of the image frame 102. After thedisplay offset 110 has been determined, the display unit 17 displays aportion 102 d of the image frame 102 according to the display offset110.

Please refer to FIG. 1B, which illustrated a portion 100 d of the imageframe 100, the display offset 110, and the portion 102 d of the imageframe 102. In this embodiment, the display unit 17 displays only theportion 100 d of the image 100 and the portion 102 d of the image frame102. Since the position of the portion 102 d to be displayed has beenadjusted based on the display offset 110 and the portion surrounding theportion 102 d will not be displayed, the image frames shown on thedisplay unit 17 are stabilized to human eyes.

It is noted that, in other embodiments, the condition of the ambientlight of the electronic device 1 can be used for determining whether thestabilization is required as well. Specifically, when the intensity ofthe image frame 102 is below a predetermined threshold (i.e. meaningthat the ambient light is weak), the image processing unit 15 will skipthe process of image stabilization due to the extraordinary amount ofnoises contained in the image frame 102. Yet in other embodiments, thecondition of an auto-focus component can be used for determining whetherthe stabilization is required as well. Particularly, when the auto-focuscomponent is functioning, the image processing unit 15 will skip theprocess of image stabilization to prevent from creating a skew imageframe.

Moreover, in other embodiments, the electronic device 1 may comprise anadditional pre-processing unit (not shown) electrically connectedbetween the image input unit 11 and the stabilization unit 155. Afterthe image input unit 11 captures the image frames 100, 102, thepre-processing unit may perform various processing (e.g. de-mosaic,de-noise, scaling, etc.) on the image frames 100, 102. Thereafter, theimage processing unit 15 and the display unit 17 work on thepre-processed image frames (not shown) instead of the image frames 100,102.

According to the above description, the electronic device 1 willdetermine a display offset for an image frame when the stabilization isdetermined required. Hence, the effect of small disturbances caused dueto undesired shaking (e.g. hand shaking) can be minimized

A second embodiment of the present invention is an electronic device 2and a schematic view of which is depicted in FIG. 2. The electronicdevice 2 comprises an image input unit 11, a sensor input unit 13, animage processing unit 25, and a display unit 17. The image processingunit 25 comprises a motion analysis unit 251, a motion determinationunit 153, and a stabilization unit 155. The motion analysis unit 251comprises a high pass filter 251 a, a variance calculation unit 251 b,and a variance comparison unit 251 c. The image input unit 11, thesensor input unit 13, the display unit 17, the motion determination unit153, and the stabilization unit 155 comprised in the electronic device 2perform similar operations as those comprised in the electronic device 1of the first embodiment. Hence, the following descriptions will befocused on the operations performed by the motion analysis unit 251 ofthe electronic device 2.

In this embodiment, the high pass filter 251 a is electrically connectedto the sensor input unit 13 and the variance calculation unit 251 b.Besides, the variance comparison unit 251 c is electrically connected tothe variance calculation unit 251 b and the motion determination unit153.

The high pass filter 251 a receives the sensor data 104 and filters thesensor data 104 into the filtered sensor data 105. In this way, thefiltered sensor data 105 contains only high frequency components of thesensor data 104; that is, the components that are related to themovement/shaking of the electronic device 2 within the time interval).The variance calculation unit 251 b receives the filtered sensor data105 from the high pass filter 105 and generates the motion variance 106by performing an arithmetic operation on at least a portion of thefiltered sensor data 105 and a plurality of weight values. The variancecomparison unit 251 c then determines a motion value 107 by comparingthe motion variance 106 with a first threshold. Afterwards, the motiondetermination unit 153 determines whether to perform stabilization onthe image frame 102 according to the motion value 107 (i.e. a value thatis related to the motion variance 106).

An example for determining the motion variance 106 and the motion value107 is given below, which, however, is not used to limit the scope ofthe present invention. In this example, the sensor data is g-sensor dataand each set of the filtered sensor data 105 is represented as (x, y,z). It is noted that the variable x denotes x-axis gravity information,the variable y denotes y-axis gravity information, and the variable zdenotes z-axis gravity information.

Next, the variance calculation unit 251 b performs the followingarithmetic operation to determine the motion variance 106. For each setof the filtered sensor data 105, the variance calculation unit 251 bcalculates a root-mean-square by the equation w=√{square root over(x²+y²+z²)} and determines the motion variance 106 by the equationw_(f)=αw_(t)+(1−α)Σw(t−n). The variable w_(f) denotes the motionvariance 106 of the image frame 102, the variable w_(t) denotes theroot-mean-square of the latest set of the filtered sensor data 105, thevariable w_(t−n) denotes the sets of the filtered sensor data 105 exceptthe latest set, the variable n is a positive integer and is the numberof filtered sensor data, and the variable α is a value between 0 and 1.

From the aforementioned equation, it is understood that the number ofthe data sets of the filtered data 105 (i.e. variable n) used forcalculating the root-mean-square and the consequent motion variance 106is adjustable. Hence, the motion variance 106 of a current image frame(e.g. the image frame 102) depends on not only the latest status thesensor input unit 13 but also the previous status of the sensor inputunit 13 over a period of time. That is, the motion variance 106 will bemore accurate when more sets of the filtered sensor data 105 areincluded during the calculation.

Following that, the variance comparison unit 251 c determines whetherthe electronic device 2 is in the condition of actual movement or in thecondition of subtle movement caused by undesired handshaking.Particularly, the variance comparison unit 251 c may determine themotion value 107 by comparing the motion variance 106 with the firstthreshold. The variance comparison unit 251 c may set the value of themotion value 107 to the integer 1 to indicate the motion variance 106 isgreater than the first threshold. On the contrary, the variancecomparison unit 251 c may set the value of the motion value 107 to theinteger 0 to indicate the motion variance 106 is equal to or smallerthan the first threshold. After the motion value 107 has beendetermined, the motion determination unit 153 can determine whether toperform stabilization on the image frame 102 according to the motionvalue 107.

In some other embodiments, the motion analysis unit 251 may omit thehigh pass filter 251 a from the motion analysis unit 251. Under thesecircumstances, the variance calculation unit 251 b determines the motionvariance 106 based on the sensor data 104 instead.

According to the above description, the electronic device 2 performs anarithmetic operation for determining the motion variance 106. Since thefiltered sensor data 105 used for determining the motion variance 106covers a period of time, the motion variance 106 and the consequentmotion value 107 become more accurate. As a result, whether to performthe stabilization on the image frame 102 can be determined in anaccurate and efficient fashion.

A third embodiment of the present invention is an electronic device 3and a schematic view of which is depicted in FIG. 3A. The electronicdevice 3 comprises an image input unit 11, a sensor input unit 13, animage processing unit 35, a display unit 17, and a storage unit 39. Theimage processing unit 35 comprises a motion analysis unit 251, a motiondetermination unit 353, and a stabilization unit 155. The motiondetermination unit 353 comprises a value determination unit 353 a and amode selection unit 353 b. The image input unit 11, the sensor inputunit 13, the display unit 17, the motion analysis unit 251, and thestabilization unit 155 comprised in the electronic device 3 performsimilar operations as those comprised in the electronic device 2 of thesecond embodiment. Hence, the following descriptions will be focused onthe operations performed by the motion determination unit 353 and thestorage unit 39 of the electronic device 3.

In this embodiment, the value determination unit 353 a is electricallyconnected to the variance comparison unit 251 c and the mode selectionunit 353 b. The mode selection unit 353 b is electrically connected tothe stabilization unit 155. The storage unit 39 is electricallyconnected to the value determination unit 353 a.

After the variance comparison unit 251 c generates the motion value 107,the value determination unit 353 a determines a motion ratio 109corresponding to the image frame 102. Please refer to FIG. 3B for theconcept of determining the motion ratio 109, wherein the horizontal axisindicates the time. The integer shown in the most right-hand side is themotion value 107 corresponding to the image frame 102. The rest integersshown in FIG. 3B are the motion values corresponding to other imageframes captured prior to the image frame 102. The motion valuescorresponding to other image frames are stored in the storage unit 39.Specifically, the value determination unit 353 a determines the motionratio 109 corresponding to the image frame 102 according to the motionvalue 107 corresponding to the image frame 102 and a plurality of motionvalues corresponding to other image frames captured prior to the imageframe 102. For example, the value determination unit 353 a can use atime window 30 of a predetermined size and select the motion valuescomprised in the time window 30 to determine the motion ratio 109. Asshown in FIG. 3B, the predetermined size of the time window 30 is tenand six of the ten motion values comprised in the time window 30 are ofvalue 1, so the value determination unit 353 a determines that themotion ratio 109 is six-tenths.

Thereafter, the mode selection unit 353 b compares the motion ratio 109with a second threshold and determines whether to perform thestabilization according to the comparison result. As an example, themode selection unit 353 b may set the second threshold to the valuethree-tenth. Under this circumstance, when the motion ratio 109 isgreater than the second threshold, the mode selection unit 353determines that stabilization of the image frame 102 is required. On thecontrary, when the motion ratio 109 is equal to or smaller than thesecond threshold, the mode selection unit 353 determines thatstabilization of the image frame 102 is not required. The determinationresult 108 of the mode selection unit 353 b will be transmitted to thestabilization unit 155 for determining the display offset 110.

In other embodiments, the motion analysis unit 251 can be replaced bythe motion analysis unit 151 in the first embodiment. In this way, thevalue determination unit 353 will determine the motion ratio 109 basedon the motion variances instead.

According to the above description, the electronic device 3 determineswhether to perform the stabilization on the image frame 102 based on themotion ratio 109. Since the motion ratio 109 reflects the ratio of themotion of the electronic device 3, whether to perform the stabilizationcan be determined more accurately and efficiently.

A fourth embodiment of the present invention is an electronic device 4and a schematic view of which is depicted in FIG. 4A. The electronicdevice 4 comprises an image input unit 11, a sensor input unit 13, animage processing unit 45, a display unit 17, and a storage unit 39. Theimage processing unit 45 comprises a motion analysis unit 251, a motiondetermination unit 353, and a stabilization unit 455. As shown in FIG.4B, the stabilization unit 455 comprises a down sample unit 455 a, afeature extraction unit 455 b, and a motion estimation unit 455 c. It isnoted that, in other embodiments, the motion analysis unit 251 may bereplaced by the motion analysis unit 151 and the motion determinationunit 353 may be replaced by the motion determination unit 153, which arereadily appreciated by those of ordinary skill in the art.

The image input unit 11, the sensor input unit 13, the display unit 17,the storage unit 39, the motion analysis unit 251, and the motiondetermination unit 353 comprised in the electronic device 4 performsimilar operations as those comprised in the electronic device 3 of thethird embodiment. Hence, the following descriptions will be focused onthe operations performed by the stabilization unit 455 of the electronicdevice 4.

In this embodiment and with reference to FIGS. 4A and 4B, the downsample unit 455 a is electrically connected to the mode selection unit353 b, the feature extraction unit 455 b, and the image input unit 11.The motion estimation unit 455 c is electrically connected to thefeature extraction unit 455 b and the storage unit 39. The down sampleunit 455 a receives the determination result 108 from the mode selectionunit 353 b, which indicates whether the stabilization on the image frame102 is required. The following description will be focused on thesituation that the stabilization is required.

The down sample unit 455 a obtains a down-sampled image frame 102 a bydown sampling the image frame 102 by a ratio. The purpose ofdown-sampling the image frame 102 is to reduce the computationcomplexity and load of the feature extraction unit 455 b and the motionestimation 455 c. Next, the feature extraction unit 455 b obtains asaliency region 102 b within the image frame 102, which may be achievedby performing a feature map searching on the down-sampled image frame102 a. During the process the feature map searching, the featureextraction unit 455 b may perform various known processing on thedown-sampled image 102 a, such as blurring, normalization, equalization,etc.

It is worth to mention that the feature extraction unit 455 b hasobtained a saliency region 100 b of the image frame 100 when processingthe image frame 100 by the similar fashion. The saliency region 100 b ofthe image frame 100 is stored in the storage unit 39. Since the imageframe 100 is treated as a reference image frame, the saliency region 100b is treated as a reference saliency region. Afterwards, the motionestimation unit 455 c determines an offset between the saliency region102 b and the saliency region 100 b and then determines the displayoffset 110 according to the offset.

As an example, the size of the image frames 100, 102 may be M-by-N, thesize of the down-sampled image frame 102 a may be P-by-Q, and the sizeof the saliency regions 100 b, 102 b may be S-by-T. It is noted that

${r = {\frac{P}{M} = \frac{Q}{N}}},$

wherein the variable r indicates the ratio used by the down sample unit455 a. Furthermore, the size of the saliency regions 100 b, 102 b issmaller than the size of the down-sampled image frame 102 a. Since theoffset is determined based on the saliency regions 100 b, 102 b, themotion estimation unit 455 c may scale the offset to the display offset110 according to the size ratio between the image frame 102 anddown-sampled image frame 102 a.

In some embodiments, the motion estimation unit 455 c may furthercomprise a coarse motion estimation unit 41 and a fine motion estimationunit 43 as shown in FIG. 4C. The coarse motion estimation unit 41determines a global offset 400 between the saliency regions 100 b, 102 bin a down-sampled resolution (i.e. P-by-Q). The fine motion estimationunit 43 then determines the display offset 110 between the saliencyregions 100 b, 102 b in a full resolution (i.e. M-by-N) according to theglobal offset 400. Particularly, the fine motion estimation unit 43 mayscale the global offset 400 by the ratio (i.e. r) used by the downsample unit 455 a to locate the position of a fine reference saliencyregion in the image frame 102. The storage unit 39 stores a finereference saliency region 100 c of the image frame 100. The fine motionestimation unit 43 then determines the display offset 110 by comparingthe fine reference saliency region of the image frame 102 and the finereference saliency region 100 c. To achieve better performance, the finemotion estimation unit 43 may divide the fine reference saliency regionof the image frame 102 into two portions (such as the left portion andright portion) and determine which portion is closer to the finereference saliency region 100 c. Next, for the portion that is closer tothe fine reference saliency region 100 c, the fine motion estimationunit 43 may continue to divide it into two exclusive sub-portions,determine which sub-portion is closer to the fine reference saliencyregion 100 c, and so on. By doing so, the display offset 110 can beefficiently determined

According to the above description, the electronic device 4 determinesthe display offset 110 by determining the offset between twosmaller-sized saliency regions 100 b, 102 b and scaling the offset intoa proper size. If the stabilization has to be more accurate, the finemotion estimation unit 43 can be adopted to determine the display offset110 by comparing two fine reference saliency regions. Hence, the displayoffset 110 can be performed in an efficient and accurate way.

A fifth embodiment of the present invention is a method for imagestabilization for use in an electronic device such as the electronicdevices 1-4. A flowchart of the method is illustrated in FIGS. 5A-5E.

First, step S51 is executed by the electronic device to capture an imageframe. Next, step S52 is executed to receive a plurality of sensor dataduring a time interval comprising a time the image frame is captured. Inother words, both the steps S51 and S52 are executed within the timeinterval. Step S53 is then executed by the electronic device todetermine a motion variance corresponding to the image frame accordingto the plurality of sensor data.

Afterwards, step S54 is executed to determine whether to performstabilization on the image frame according to the motion variance. Ifthe step S54 determines that the stabilization is required, steps S55,S56, S57 are then executed. If the step S54 determines that thestabilization is not required, steps S58, S56, S57 are then executed.

When the stabilization is required, step S55 is executed to determine amotion offset of a saliency region of the image frame to a referencesaliency region of a reference image frame. It is noted that thereference image frame is captured prior to the image frame and is storedin a memory unit within the electronic device. On the contrary, step S58is executed to set the motion offset to 0 when the stabilization is notrequired. Following that, step S56 is executed for cropping a portion ofthe image frame according to the motion offset. Finally, step S57 isexecuted to display the portion of the image frame on a display unit ofthe electronic device.

In some embodiments, the step S53 may be realized by the steps S531,S533, and S535 illustrated in FIG. 5B and the step S54 may be realizedby the steps S541, S543, S545, and S547 as shown in FIG. 5C.

Particularly, step S531 is executed to filter the plurality of sensordata with a high pass filter by the electronic device. Next, step S533is executed to apply a plurality of weight values to the plurality offiltered sensor data. Afterwards, step S535 is executed for calculatingthe motion variance corresponding to the image frame by performing anarithmetic operation on the plurality of the filtered sensor data andcorresponding weight values.

In step S541, the electronic device determines a motion valuecorresponding to the image frame by comparing the motion variance with afirst predetermined threshold. Subsequently, step S543 is executed todetermine a motion ratio corresponding to the image frame according tothe motion value of the image frame and a plurality motion valuescorresponding to a predetermined number of image frames captured priorto the image frame. It should be aware that the plurality of motionvalues corresponding to the predetermined number of image frames areaccessed from a storage unit. The electronic device then executes stepS545 to compare the motion ratio with a second predetermined threshold.Next, step S547 is executed for determining whether to perform thestabilization on the image frame according to the comparison result. Forexample, when the comparison result of the step S545 indicates that themotion ration is greater than the second predetermined threshold, thestep S547 may determine that the stabilization has to be performed. Onthe contrary, when the comparison result of the step S545 indicates thatthe motion ration is not greater than the second predeterminedthreshold, the step S547 may determine that the stabilization is notrequired.

In some embodiments, the step S55 may be realized by the steps S551-S558illustrated in FIG. 5D. Specifically, steps S551-S553 are executed fordetermining a global offset in a lower resolution, while steps S554-S558are executed for determining the motion offset in an original resolutionaccording to the global offset. It is understood that the lowerresolution is lower than the original resolution.

In step S551, the electronic device down-samples the image frame by aratio. Next, step S552 is executed for performing a feature extractionon the down-sampled image frame to obtain the first saliency region. Theelectronic device than executes step S553 to compare the first saliencyregion with the reference saliency region to obtain the global offset.

In step S554, the electronic device up-scales the global offset by aratio, such as the ratio in step S551. Following that, step S555determines a position of the first saliency region within the imageframe according to the up-scaled global offset. Next, step S556 comparesthe first saliency region at the determined position with the referencesaliency region. In step S557, the electronic device adjusts theposition of the first saliency region according to the comparisonresult. Finally, the step S558 is executed by the electronic device todetermine the motion offset according to the adjusted position of thefirst saliency region.

Yet in some embodiments, the step S56 may be realized by the steps S561,S563, and S565 illustrated in FIG. 5E. Specifically, step S561 isexecuted by the electronic device to locate a position of a displayportion of a previous image frame preceding to the image frame accordingto the motion offset. Next, step S563 is executed for determining aposition of the portion of the image frame according to the motionoffset. Finally, step S565 is executed by the electronic device forcropping the adjusted region from the image frame.

In addition to the aforesaid steps, the fifth embodiment can execute allthe operations and functions set forth for the electronic devices 1-4 inthe first to the fourth embodiments. How the fifth embodiment executesthese operations and functions will be readily appreciated by those ofordinary skill in the art based on the explanation of the first to thefourth embodiments, and thus will not be further described therein.

A sixth embodiment of the present invention is a method for imagestabilization for use in an electronic device such as the electronicdevices 1-4. A flowchart of the method is illustrated in FIG. 6.

First, step 61 is executed by the electronic device for receiving aplurality of motion data, wherein the at least a portion of theplurality of motion data corresponding to a first image frame. Next,step S63 is executed for determining a motion degree corresponding tothe first image frame according to the portion of motion data.

In some embodiments, the step S63 can be realized by applying a weightvalue to each of the portion of the motion data, calculating a motionvariance corresponding to the first image frame according to the portionof the motion data and the corresponding weight values, and calculatingthe motion degree corresponding to the first image frame according tothe motion variance corresponding to the first image frames and aplurality of pre-stored motion variance corresponding to a plurality ofsecond image frames displayed prior to the first image frame.

After the step S63, step S65 is executed by the electronic device fordetermining a display offset between the first image frame to areference image frame according to the motion degree. In someembodiments, the step S63 can be realized by performing a featureextraction to obtain the first saliency region in the first image frameand determining whether the motion degree corresponding to the firstimage frame exceeds a predetermined threshold. It is noted that thefirst saliency region and the reference saliency region comprisessubstantially identical scene.

When the motion degree exceeds the predetermined threshold, step 63further executes a step for determining the display offset according toa position offset between a first saliency region of the first imageframe and a reference saliency region of the reference image frame. Onthe contrary, when the motion degree does not exceed the predeterminedthreshold, step 63 further set the display offset to 0.

After the step S65, step S67 is executed by the electronic device forproviding a portion of the first image frame for display according tothe display offset. It should be understood that the reference imageframe is displayed prior to the first image frame and is stored in astorage unit. In some embodiments, the step S63 can be realized byobtaining a starting position of a display window in the reference imageframe, moving the starting position of the display window in the firstimage frame according to the display offset, and cropping the portion ofthe first image frame within the display window.

In addition to the aforesaid steps, the fifth embodiment can execute allthe operations and functions set forth for the electronic devices 1-4 inthe first to the fourth embodiments. How the fifth embodiment executesthese operations and functions will be readily appreciated by those ofordinary skill in the art based on the explanation of the first to thefourth embodiments, and thus will not be further described therein.

According to the above embodiments, an electronic device that adopts thepresent invention will determine motion information (e.g. theaforementioned motion variance, motion value, and/or motion degree) ofan image frame according to sensor data that are received within a timeinterval comprising a time the image frame is captured, determinewhether stabilization is required, and determine a display offset forthe image frame. Various kinds of motion information are provided by thepresent invention so that whether the stabilization is required can bedetermined more accurately and efficiently. In addition, various kindsof techniques for performing motion estimation are provided by thepresent invention so that the display offset can be determined moreaccurately and efficiently. Hence, the effect of small disturbancescaused due to undesired shaking (e.g. hand shaking) can be minimized

The above disclosure is related to the detailed technical contents andinventive features thereof. People skilled in this field may proceedwith a variety of modifications and replacements based on thedisclosures and suggestions of the invention as described withoutdeparting from the characteristics thereof. Nevertheless, although suchmodifications and replacements are not fully disclosed in the abovedescriptions, they have substantially been covered in the followingclaims as appended.

What is claimed is:
 1. A method for image stabilization for use in anelectronic device, comprising: capturing an image frame; receiving aplurality of sensor data during a time interval comprising a time theimage frame is captured, determining a motion variance corresponding tothe image frame according to the plurality of sensor data; determiningwhether to perform a stabilization on the image frame according to themotion variance; and in response to the stabilization being determinedto be performed: determining a motion offset of a first saliency regionof the image frame to a reference saliency region of a reference imageframe; cropping a portion of the image frame according to the motionoffset; and displaying the portion of the image frame on a display unitof the electronic device; wherein the reference image frame is capturedprior to the image frame and is stored in a memory unit within theelectronic device.
 2. The method of claim 1, wherein the step ofdetermining the motion variance comprises: filtering the plurality ofsensor data with a high pass filter; applying a plurality of weightvalues to the plurality of filtered sensor data; calculating the motionvariance corresponding to the image frame by performing an arithmeticoperation on the plurality of the filtered sensor data and correspondingweight values.
 3. The method of claim 2, wherein the step of determiningwhether to perform the stabilization comprises: determining a motionvalue corresponding to the image frame by comparing the motion variancewith a first predetermined threshold; determining a motion ratiocorresponding to the image frame according to the motion value of theimage frame and a plurality motion values corresponding to apredetermined number of image frames captured prior to the image frame,wherein the plurality of motion values corresponding to thepredetermined number of image frames are accessed from a storage unit;comparing the motion ratio with a second predetermined threshold; anddetermining whether to perform the stabilization on the image frameaccording to the comparison result.
 4. The method of claim 1, whereinthe step of determining of the motion offset comprises: determining aglobal offset in a lower resolution; and determining the motion offsetin an original resolution according to the global offset; wherein thelower resolution is lower than the original resolution.
 5. The method ofclaim 4, wherein the step of determining of the global offset comprises:down-sampling the image frame by a ratio; performing a featureextraction on the down-sampled image frame to obtain the first saliencyregion; and comparing the first saliency region with the referencesaliency region to obtain the global offset.
 6. The method of claim 4,wherein the step of determining of the motion offset in the originalresolution comprises: up-scaling the global offset by a ratio;determining a position of the first saliency region within the imageframe according to the up-scaled global offset; comparing the firstsaliency region at the determined position with the reference saliencyregion; adjusting the position of the first saliency region according tothe comparison result; and determining the motion offset according tothe adjusted position of the first saliency region.
 7. The method ofclaim 1, wherein the step of cropping of the portion of the image framecomprises: locating a position of a display portion of a previous imageframe preceding to the image frame according to the motion offset;determining a position of the portion of the image frame according tothe motion offset; and cropping the adjusted region from the imageframe.
 8. An electronic device, comprising: an image input unit,configured to capture a first image frame; a sensor input unit,configured to receive a plurality of sensor data, the plurality ofsensor data is received by the sensor input unit during a time intervalcomprising a time the first image frame is captured; and an imageprocessing unit, configured to process the first image frame andcomprising: a motion analysis unit, configured to determine a motionvariance corresponding to the first image frame according to theplurality of sensor data; a motion determination unit, configured todetermine whether to perform a stabilization on the first image frameaccording to the motion variance corresponding to the first image frame;and a stabilization unit, configured to determine a display offset ofthe first image frame with respect to a reference image frame accordingto the stabilization determination; and a display unit, configured todisplay a portion of the first image frame according to the displayoffset.
 9. The electronic device of claim 8, wherein the motion analysisunit comprises: a variance calculation unit, configured to determine themotion variance by performing an arithmetic operation on a portion ofthe plurality of sensor data and a plurality of weight values; and avariance comparison unit, configured to compare the motion variance witha first threshold.
 10. The electronic device of claim 9, wherein themotion analysis unit further comprises a high pass filter configured tofilter the plurality of sensor data.
 11. The electronic device of claim8, wherein the motion determination unit determination unit furthercomprises: a value determination unit, configured to determine a motionratio corresponding to the first image frame according to a motion valuecorresponding to the first image frame and a plurality of motion valuescorresponding to a plurality of second image frames captured prior tothe first image frame; and a mode selection unit, configured to comparethe motion ratio with a second predetermined threshold and determinewhether to perform the stabilization according to the comparison result.12. The electronic device of claim 11, further comprises a storage unitconfigured to store the plurality of motion values corresponding to theplurality of second image frames.
 13. The electronic device of claim 8,wherein the stabilization unit further comprises: a down sample unit,configured to down sample the first image frame by a ratio; a featureextraction unit, configured to obtain a first saliency region within thefirst image frame; a motion estimation unit, configured to determine thedisplay offset according to an offset between the first saliency regionand a reference saliency region within a reference image frame, whereinthe reference image frame is captured prior to the first image frame.14. The electronic device of claim 13, wherein the motion estimationunit further comprises: a coarse motion estimation unit, configured todetermine a global offset between the first saliency region and thereference saliency region in a down-sampled resolution; and a finemotion estimation unit, configured to determine the display offsetbetween the first saliency region and the reference saliency region in afull resolution according to the global offset wherein the down-sampledresolution is lower than the full resolution.
 15. A method for imagestabilization for use in an electronic device, comprising: receiving aplurality of motion data, at least a portion of the plurality of motiondata corresponding to a first image frame; determining a motion degreecorresponding to the first image frame according to the portion ofmotion data; determining a display offset between the first image frameto a reference image frame according to the motion degree; and providinga portion of the first image frame for display according to the displayoffset; wherein the reference image frame is displayed prior to thefirst image frame and is stored in a storage unit.
 16. The method ofclaim 15, wherein the step of determining of the motion degreecomprises: applying a weight value to each of the portion of the motiondata; calculating a motion variance corresponding to the first imageframe according to the portion of the motion data and the correspondingweight values; calculating the motion degree corresponding to the firstimage frame according to the motion variance corresponding to the firstimage frames and a plurality of pre-stored motion variance correspondingto a plurality of second image frames displayed prior to the first imageframe.
 17. The method of claim 15, wherein the step of determining ofthe display offset further comprises: determining whether the motiondegree corresponding to the first image frame exceeds a predeterminedthreshold; in response to the motion degree exceeding the predeterminedthreshold, determining the display offset according to a position offsetbetween a first saliency region of the first image frame and a referencesaliency region of the reference image frame; wherein the first saliencyregion and the reference saliency region comprises substantiallyidentical scene.
 18. The method of claim 17, wherein the step ofdetermining of the display offset further comprises performing a featureextraction to obtain the first saliency region in the first image frame.19. The method of claim 17, wherein the display offset is set to 0 inresponse to the motion degree corresponding to the first image frame notexceeding the predetermined threshold.
 20. The method of claim 15,wherein the providing of the portion of the first image frame furthercomprises: obtaining a starting position of a display window in thereference image frame; moving the starting position of the displaywindow in the first image frame according to the display offset;cropping the portion of the first image frame within the display window.